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Abstract 

The  objective  of  this  demonstration  was  to  compare  the  discrimination  potential  of  three 
dynamic  surveys  among  themselves  and  with  static  grid  data  over  common  targets  at  the  CT-3A 
Bombing  target,  Former  Badlands  Bombing  Range,  South  Dakota.  This  demonstration 
leveraged  an  ordnance  and  explosive  removal  action  funded  by  the  Corps  of  Engineers,  whose 
objective  was  to  remove  all  items  larger  than  a  40mm  projectile.  We  analyzed  data  from  part  of 
the  bombing  target  with  the  goal  of  identifying  particular  targets  of  interest  that  could  be 
discriminated  from  the  plethora  of  munitions  debris  that  is  present  at  this  locale.  Recovered 
targets  of  interest  were  unfortunately  few  in  number,  but  include  (i)  four  20mm  projectiles,  (ii) 
six  2.25inch  aluminum  sub-caliber  aircraft  rocket  warheads,  and  (iii)  one  ferrous  2.75inch 
folding-fin  aircraft  rocket  warhead. 

Data  from  multiple  geophysical  systems  were  analyzed  using  UX- Analyze.  The  data  sets 
include  (i)  electromagnetic  survey  data  acquired  using  a  man-portable  litter  configuration  and  a 
vehicle-towed  array,  (ii)  electromagnetic  data  acquired  in  a  grid  over  individual  targets,  and  (iii) 
magnetic  data  acquired  using  a  man-portable  two-sensor  system.  UX- Analyze  provides  the 
necessary  infrastructure  to  systematically  identify  and  extract  anomalies  from  the  dataset, 
execute  the  characterization  routines,  and  perform  target  classification.  It  is  integrated  into  Oasis 
montaj™,  a  commercial  data  processing  and  visualization  software  suite.  The  inversion  routines 
within  UX- Analyze  assume  a  dipolar  source  and  derive  the  best  set  of  induced  dipole  model 
parameters  that  account  for  the  spatial  variation  of  the  signal  as  the  sensor  is  moved  over  the 
object. 

Discriminating  the  20mm  projectiles  and  2.25inch  aluminum  warheads  from  the  plethora  of 
munitions  debris  responses  based  on  their  inverted  shape  parameters  proved  challenging.  Even 
though  the  litter-based  EMI  sensor  was  carried  along  transects  spaced  0.5m  apart,  many 
signatures  were  spatially  limited  and  observed  on  only  a  single  profile.  Additionally,  as 
measured  by  the  dynamic  survey  platforms,  the  anomalies  could  not  be  well  represented  by  a 
dipole  model.  The  mean  fit  error  is  33%  (standard  deviation  of  27.3)  for  the  litter-carried  data 
and  30%  (standard  deviation  of  17.5)  for  the  towed  array  data.  We  anticipated  that  the  limited 
spatial  views  of  individual  targets,  small  spatial  footprint,  and  low-amplitude  signatures  hindered 
the  EMI  characterization  routines’  ability  to  estimate  accurate  size,  shape,  and  depth  estimates. 

In  stark  comparison  to  the  litter-carried  and  towed-array  EMI  data,  the  gridded  EMI  data 
acquired  in  a  cued  fashion  was  well  represented  by  a  dipole  model.  For  commonly  selected 
targets,  the  mean  dipole  fit  error  was  5%  for  the  cued  system  compared  to  >23%  for  the  dynamic 
surveys.  Additionally,  with  the  exception  of  one  horizontal  37mm  buried  at  30cm,  the  inverted 
shape  parameters  for  symmetric  targets  in  the  geophysical  prove  out  appear  symmetric. 

Although  the  target  population  is,  as  a  rule,  too  small  to  be  properly  characterized  by  inverting 
individual  signatures  as  measured  by  a  dynamic  platform,  a  significant  number  of  the  munitions 
debris  items  can  be  screened  by  setting  appropriate  thresholds  if  specific  goals  are  established  for 
the  site.  Ninety  seven  percent  of  the  anomalies,  for  example,  possess  peak  responses  that  are  less 
than  0.8  times  that  of  the  2.75inch  ferrous  warhead.  Approximately  thirty  percent  of  the 
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anomalies  possess  amplitudes  that  are  less  than  0.5  times  that  measured  for  the  2.25inch 
aluminum  warheads.  Using  a  less  conservative  threshold  of  0.8  times  that  of  the  2.25inch 
warhead,  we  find  that  sixty  percent  of  the  anomalies  could  be  classified  as  too  small. 
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Badlands  Bombing  Range,  South  Dakota 


1.  Introduction 


1.1  Background 

The  objective  of  this  ESTCP  project  is  to  further  feature-based  discrimination  approaches  through 
demonstrations  at  live  sites  and  by  embedding  analysis  routines  into  a  commercial  software 
package  for  use  by  others.  Major  tasks  include  (i)  transitioning  target  characterization  and 
classification  algorithms  for  magnetic  and  electromagnetic  data  to  a  commercial  product,  and  (ii) 
establishing  the  efficacy  and  cost  effectiveness  of  the  decision-making  approach  through 
demonstrations  at  sites  with  varying  site  conditions  and  survey  goals. 

Feature-based  characterization  and  classification  schemes  have  improved  discrimination 
performance  in  some  demonstrations.  These  algorithms,  however,  are  not  readily  available  to  the 
user  community  and  have  had  limited  exposure  to  data  acquired  under  'production-imposed' 
constraints.  This  program  is  designed  to  address  the  availability  problem  as  well  as  further  test  the 
ability  to  improve  decisions  using  features  -  or  model  parameters  -  derived  from  field  data. 

Our  technical  approach  promotes  the  selection  of  potential  UXO  targets  using  quantitative 
evaluation  criteria  and  transparent  decision-making  processes.  As  such,  we  developed  an  analysis 
framework  within  Oasis  montaj™  and  embedded  previously  developed  analysis  algorithms.  The 
analysis  algorithms  provide  quantitative  evaluation  criteria;  namely,  target  characterization 
followed  by  statistical  classification.  Transparency  is  achieved  by  leveraging  the  professional, 
flexible,  and  visual  computing  environment  inherent  in  Oasis  montaj™,  a  commercial  geophysical 
data  processing  and  visualization  package. 

This  demonstration  leveraged  an  ongoing  OE  removal  action  at  the  Badlands  Bombing  Range 
(BBR)  in  order  to  evaluate  the  merits  of  using  quantitative,  physics-based  decision  criteria  encoded 
in  UX- Analyze™  for  the  purpose  of  anomaly  discrimination.  Documented  ordnance  items  found 
at  the  site  include  20mm  projectiles,  2.25  inch  rockets,  2.75  inch  rockets,  40mm  anti-aircraft 
projectiles,  75mm  projectiles,  60mm  mortars,  incendiary  bomblets,  1001b  practice  bombs  and 
2501b  and  5001b  bombs  (Parsons,  2004). 

1.2  Objectives  of  the  Demonstration 

This  demonstration  has  two  primary  objectives,  (i)  compare  and  contrast  the  discrimination 
potential  of  three  productions  surveys  over  common  targets  (two  EMI  and  one  magnetic  survey), 
and  (ii)  investigate  effects  of  data  quality  on  inverted  fit  results  by  comparing  data  acquired  in  a 
gridded,  cued  mode  to  standard  dynamic  survey  modes. 
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1.3  Regulatory  Drivers 


The  Senate  Report  (Report  106-50),  pages  291-293,  accompanying  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2000  (Public  Law  106-65),  included  a  provision  entitled 
“Research  and  development  to  support  unexploded  ordnance  clearance,  active  range  unexploded 
ordnance  clearance,  and  explosive  ordnance  disposal.”  This  provision  requires  the  Secretary  of 
Defense  to  submit  to  the  Congressional  defense  committees  a  report  that  gives  a  complete  estimate 
of  the  current  and  projected  costs,  to  include  funding  shortfalls,  for  UXO  response  at  active 
facilities,  installations  subject  to  base  realignment  and  closure  (BRAC),  and  formerly  used  defense 
sites  (FUDS). 

The  following  statements  are  taken  verbatim  out  of  the  DoDs  2001  Report  to  Congress: 

“Decades  of  military  training,  exercises,  and  testing  of  weapons  systems  has  required  that  we 
begin  to  focus  our  response  on  the  challenges  of  UXO.  Land  acreage  potentially  containing 
UXO  has  grown  to  include  active  military  sites  and  land  transferring  or  transferred  for  private 
use,  such  as  Base  Realignment  and  Closure  (BRAC)  sites  and  Formerly  Used  Defense  Sites 
(FUDS).  DoD  responsibilities  include  protecting  personnel  and  the  public  from  explosive 
safety  hazards;  UXO  site  cleanup  project  management;  ensuring  compliance  with  federal, 
state,  and  local  laws  and  environmental  regulations;  assumption  of  liability;  and  appropriate 
interactions  with  the  public. 

. . .  Through  limited  experience  gained  in  executing  these  activities,  it  has  become  increasingly 
clear  that  the  full  size  and  extent  of  the  impact  of  sites  containing  UXO  is  yet  to  be  realized.  ... 
DoD  has  completed  an  initial  baseline  estimate  for  UXO  remediation  cost.  This  report 
provides  a  UXO  response  estimate  in  a  range  between  $106.9  billion  and  $391  billion  in 
current  year  [2001]  dollars.  ...Technology  discovery,  development,  and  commercialization 
offers  some  hope  that  the  cost  range  can  be  decreased.  . . . 

...  Objective:  Develop  standards  and  protocols  for  navigation,  geo-location,  data  acquisition 
and  processing,  and  performance  of  UXO  technologies. 

•  Standard,  high  quality  archived  data  are  needed  for  optimal  data  processing  of 
geophysical  data,  re-acquisition  for  response  activities,  quality  assurance,  quality  control,  and 
review  by  all  stakeholders.  In  addition  standards  and  protocols  are  required  for  evaluating 
UXO  technology  performance  to  aid  in  selecting  the  most  effective  technologies  for  individual 
sites. 

•  Standard  software  and  visualization  tools  are  needed  to  provide  regulatory  and  public 
visibility  to  and  understanding  of  the  analysis  and  decision  process  made  in  response 
activities.  ” 
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1.4  Stakeholder/End-User  Issues 


The  stakeholders  and  end-users  of  this  data  processing  and  analysis  technology  include  private 
contractors  that  conduct  geophysical  investigations  in  support  of  UXO  clean  up  programs  and 
governmental  employees  that  provide  technical  oversight.  This  demonstration  introduces  the 
stakeholders  and  end-users  to  data  products  associated  with  this  analysis  approach  and  to  the 
inherent  transparency  of  the  decision-making  process. 

2.  Technology  Description 

2.1  Technology  Developments  and  Application 

2.1.1  Data  Acquisition  System  Hardware 

Data  from  four  geophysical  systems  were  analyzed.  AETC  personnel  acquired  EM61  Mkll  data 
using  a  man-portable  litter  configuration  for  the  CT-3A  bombing  target  field  survey  and  a  cued 
configuration  for  individual  targets.  For  the  field  survey,  the  sensor  were  carried  in  a  litter  mode 
and  augmented  with  an  inertial  navigation  unit  and  RTK  DGPS  sensors  (Figure  1).  Magnetic 
data  were  acquired  using  NRLs  man-portable  magnetic  adjunct.  This  system  consists  of  two 
cesium-vapor  magnetometers  mounted  on  either  side  of  wheeled  assemblage.  It  integrates  with 
RTK  DGPS  for  spatial  registration. 


DGPS 


/ 


Figure  1.  Magnetometer  (left)  and  EMI  (right)  systems. 


Cued  EM61  MKII  data  were  acquired  over  twenty  individual  targets  using  a  6x5  point  rectangle 
grid  (nodes  separated  by  20.3cm  and  40.6cm  respectively  resulting  in  a  1.0x1. 6  m  area).  The 
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grid  was  elevated  20  cm  above  the  ground,  and  the  EM61  was  positioned  directly  on  the  grid, 
without  wheels.  Sensor  location  was  established  by  lining  up  cross  hairs  on  the  EM61  bottom 
coil  with  grid  line  intersections  (Figure  2). 


Figure  2.  EM61  sensor  during  cued  data  collection. 

Parsons  also  acquired  EM61  Mkll  data  using  a  three-coil  towed  array.  The  coils  are  spaced 
0.91m  (three  feet)  apart  (Figure  3). 


Figure  3.  Parsons  three-coil  EM61  array  deployed  at  the  Badlands  Bombing  Range. 
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2.1.2  UX- Analyze  Software 

The  anomaly  characterization  algorithms  used  here  assume  a  dipolar  source  and  derive  the  best 
set  of  induced  dipole  model  parameters  that  account  for  the  spatial  variation  of  the  signal  as  the 
sensor  is  moved  over  the  object.  The  model  parameters  are  target  X,Y  location  and  depth,  three 
dipole  response  coefficients  corresponding  to  the  principle  axes  of  the  target  (EMI  only),  and  the 
three  angles  that  describe  the  orientation  of  the  target.  The  size  of  the  target  can  be  estimated 
using  empirical  relationships  between  either  the  dipole  moment  for  magnetic  data  or  the  sum  of 
the  targets’  response  coefficients.  Cylindrical  objects  like  most  UXO  have  one  large  coefficient 
and  two  smaller,  equal  coefficients.  Plate-like  objects  nominally  have  two  large  and  one  small 
coefficient. 

UX- Analyze  was  developed  to  facilitate  efficient  UXO  data  analysis  within  the  Oasis  montaj™ 
environment.  It  consists  of  multiple  databases,  custom  graphical  interfaces,  and  data 
visualizations.  UX- Analyze  provides  the  infrastructure  to  systematically  identify  and  extract 
anomalies  from  the  dataset,  call  the  characterization  routines,  store  the  fitted  source  parameters 
for  each  anomaly,  perform  target  classification,  and  document  the  analysis.  Once  the  analysis  is 
complete,  individual  images  for  each  anomaly  can  be  automatically  produced  for  documentation 
purposes. 


2.2  Previous  Testing  of  the  Technology 

We  have  performed  preliminary  tests  of  the  Demonstration  data  analysis  and  classification 
technology  during  algorithm  development.  A  number  of  the  user  graphical  interfaces  and 
dialogue  boxes  are  briefly  described  below.  Each  of  these  capabilities  was  tested  prior  to  the 
demonstration. 

2.2.1  UX- Analyze 

UX- Analyze  allows  users  to  systematically  identify,  extract,  edit,  and  store  data  around 
individual  anomalies.  It  provides  efficient  data  structures  and  access  for  the  analysis  algorithms, 
stores  the  fitted  parameters,  and  allows  for  multiple  data  types  and  surveys.  This  module  is  the 
interface  between  Oasis  montaj  and  the  demonstration  analysis  software  (Figure  4). 
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Oasis  montaj  -  C:\projects\3112_ESTCP_20021D\Sample  Pata\3PGV\JPGV_Areal\EM\test.gws 


File  Edit  GX  Data  Profile  Map  Coordinates  Utility  X-Utility  Grid  Mapping  DAP  UX Analyze  Window  Help 


^JSJXJ 


Cell  |35.62 

For  Help,  press  FI  [Data  |m:  641412.046. 4309746.37 


Figure  4.  Screen  snapshots  showing  the  UX-Analyze  interface. 


2.2.2  Characterization  Modules 


Characterization  routines  for  magnetic  and  EMI  data  have  been  integrated  with  UX-Analyze 
framework.  These  3-D  routines  include  graphic  displays  and  controls  that  allow  the  user  to 
manually  select,  filter,  and  invert  each  anomaly  individually  (Figure  5)  or  in  batch.  The  derived 
model  parameters  are  stored  in  a  master  target  database.  The  inversion  routines  were  previously 
developed  by  AETC  Incorporated  for  the  MTADS  Data  Analysis  System  (DAS)  under  funding 
from  ESTCP  and  SERDP  (Barrow  and  Nelson,  1998;  ESTCP  Report  199526).  The  MTADS 
DAS  codes  were  prototyped  using  the  Interactive  Development  Language  (IDL). 

Algorithm  equivalency  tests  were  conducted  to  verify  that  the  C-based  inversion  routines 
embedded  in  Oasis  montaj  produce  identical  performances  as  the  prototyped  IDL  formulations. 
Magnetic  and  EMI  data  were  synthesized  for  forty-nine  sources  that  have  a  unique  combination 
of  inclination  and  declination  but  constant  moment  and  depth  of  burial.  To  ensure  that  each 
routine  received  the  exact  same  input  for  each  anomaly,  we  extracted  data  samples  around  each 


6 


anomaly  individually  once  and  then  used  the  extracted  data  subsets  as  input  for  both.  The  data 
sets  were  inverted  using  both  codes  and  compared.  No  significant  differences  were  observed. 


Figure  5.  Screen  snapshots  showing  the  user  interface  during  data  inversion.  In  this  figure,  the 
measured  data  is  shown  in  the  upper  left  map,  the  model  parameters  (fitted  results)  are  displayed 
in  the  lower  center  window,  and  the  forward  model  generated  using  the  model  parameters  is 
shown  in  the  upper  right  map. 

2.2.3  Classification  Module 

Classifying  objects  as  UXO  or  non-UXO  based  on  inverted  target  features  is  challenging  due  to 
imperfect,  non-unique  feature  estimates  and  potentially  non-separable  feature  classes.  The 
imperfect  and  non-unique  feature  estimates  result  largely  from  distortions  in  the  spatial 
distribution  of  the  anomaly.  The  distortions  in  turn  are  commonly  caused  by  (i)  errors  in  the 
spatial  registration  of  the  measured  data  samples,  and/or  (ii)  nearby  geologic  or  environmental 
signatures  not  associated  with  the  source  in  question.  As  a  result,  separating  the  fitted 
parameters  into  distinct  and  disparate  classes  is  non-trivial.  For  this  demonstration,  we  utilized 
the  generalized  likelihood  ratio  test  (GLRT)  -  a  statistical  classification  method  that  has  proved 
promising  during  the  recent  past  (Figure  6). 
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UXO/Clutter  target  classification 


UXO/Clutter  target  classification 


Target  database  file  | arealfeature .gdb 
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d 
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“"31 
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3 
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d 
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d 

Feature  channel  10  (Not  used  if  blank) 

d 

OK  |  Cancel  |  Help  | 


Figure  6.  Screen  snapshots  showing  the  user  dialogue  interfaces  that  call  the  classification 
routines. 


2.2.4  Data  Analysis  Documentation 

UX- Analyze  produces  individualized  anomaly  reports,  one  for  each  anomaly,  to  document  the 
decision  process  for  each  anomaly  (Figure  7).  In  each  plot,  the  measured  data  is  graphically 
displayed  next  to  the  modeled  data.  The  model  parameters  are  listed  in  the  middle  of  each  page, 
and  a  profile  extracted  along  the  transect  that  passes  closest  to  the  dipoles  location  -  as  estimated 
by  the  inversion  routine  -  is  located  at  the  bottom.  The  positions  of  individual  measurements  are 
superimposed  on  the  maps. 

Essentially,  the  anomaly  plots  graphically  provide  an  intuitive  confidence  measure.  If  the 
measured  and  modeled  data  are  indistinguishable,  the  reviewer  can  have  confidence  that  the 
estimated  source  parameters  are  approximately  correct.  If  the  two  maps  are  do  not  resemble 
each  other,  however,  it  tells  us  that  the  source  in  question  (i)  cannot  be  represented  well  using  a 
point  dipole  source,  (ii)  is  not  isolated,  (iii)  does  not  have  sufficient  signal-to-noise  ratio,  or  (iv) 
was  not  properly  sampled  (spatially  or  temporally).  In  any  case,  if  the  two  maps  are  dissimilar 
the  inverted  model  parameters  are  most  likely  not  correct. 
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Figure  7.  To  document  the  analysis,  UX-Analyze  generates  a  one-page  summary  for  each 
anomaly.  In  the  anomaly  summaries  shown  above,  the  measured  data  is  shown  in  the  upper  left 
hand  corner,  the  inverted  model  parameters  in  the  middle  left,  the  forward  model  in  the  upper 
rights,  and  a  profile  in  the  lower  left  corner.  This  layout  was  selected  to  provide  insight  into  the 
confidence  of  the  analysis  and  conclusions.  EMI  data  for  the  anomaly  are  shown  in  the  left 
summary,  and  magnetic  data  on  the  right. 


2.3  Factors  Affecting  Cost  and  Performance 

The  analysis  approach  adopted  here  utilizes  the  spatial  distribution  of  the  measured  magnetic  or 
EMI  signatures.  As  such,  it  requires  high  signal-to-noise  data  that  possess  a  high  degree  of  spatial 
precision  across  the  footprint  of  the  anomaly.  The  costs  to  acquire  data  that  will  support 
discrimination  decisions  are  higher  than  that  required  if  the  goal  is  only  to  detect  the  presence  of  an 
object.  The  analysis  costs  are  also  higher  if  attempts  are  made  to  quantitatively  discriminate 
relative  than  only  to  detect. 

The  factors  affecting  acquisition  costs  relate  to  particulars  of  the  sensing  system,  spatial 
registration  system,  the  target  objectives,  and  the  site  environment.  Although  these  costs  are  not 
the  focus  of  this  demonstration,  they  are  very  important  to  the  ultimate  transferability  of  this 
approach. 
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The  factors  affecting  analysis  time  include  are  significantly  affected  by  (i)  the  degree  to  which  the 
anomalies  are  spatially  separated,  (ii)  the  number  of  anomalies,  and  (iii)  the  amount  of  geologic- 
related  signatures  that  possess  similar  wavelengths  as  the  targeted  signatures.  The  data  density  is 
also  a  factor,  but  only  marginally  so  compared  to  the  factors  listed  above. 

Discrimination  performance  is  measured  by  our  ability  to  characterize  and  classify  one  object  from 
another.  The  factors  that  affect  performance,  therefore,  relate  to  the  similarity  (in  feature  space) 
between  the  sought-after  object  versus  the  clutter,  our  ability  to  accurately  measure  the  responses, 
the  presence  of  signatures  that  spatially  interfere  or  otherwise  compete  with  the  UXOs  response,  as 
well  as  our  ability  to  quantitatively  characterize  and  classify  the  source  objects.  Many  of  these 
factors  are  not  under  our  direct  control. 


2.4  Advantages  and  Limitations  of  the  Technology 

This  analysis  approach  uses  spatially  referenced  geophysical  data  to  estimate  target  parameters 
for  each  anomaly.  This  has  an  inherent  advantage  over  ancillary  analysis  methods  that  are 
commonly  used.  Due  to  a  lack  of  analysis  routines  available,  many  contractors  make  UXO  and 
non-UXO  declarations  based  on  anomaly  amplitude,  half  width,  spatial  footprint,  or  overall 
‘look’.  These  characterization  methods  are  sensitive  to  the  targets’  orientation  and  depth  of 
burial  (or  distance  from  the  sensor).  The  methodology  demonstrated  here  separates  the  measured 
signatures  into  that  which  is  inherent  to  the  target,  and  that  which  is  related  to  the  geometry  of 
the  problem  (such  as  distance  to  sensor  and  orientation).  The  fitted  parameters  that  are  inherent 
to  the  target  itself  are  used  to  classify  the  unknown  object. 

The  primary  advantage,  therefore,  is  the  potential  for  discriminating  between  UXO  and  non 
UXO-like  objects  based  upon  geophysical  survey  data.  This  is  in  contrast  to  simply  identifying 
the  location  of  anomalies  from  the  geophysical  survey  data.  Results  from  past  demonstration 
have  shown  that  that  some  discrimination  is  possible  using  magnetic  and  electromagnetic  data 
(Robitaille  et  al.,  1999).  Magnetic  discrimination  is  based  primarily  on  the  apparent  fitted  dipole 
size  (or  scaled  dipole  moment).  Using  EMI  data,  increased  discrimination  performance  can 
sometimes  be  achieved  by  utilizing  estimated  shape  information.  If  successful  discrimination 
capabilities  can  be  achieved,  significant  excavation  savings  can  be  realized  by  leaving  the  non¬ 
hazard  clutter  items  unearthed. 

This  is  not  to  say,  however,  that  the  data  analysis  technology  being  demonstrated  will  solve  the 
UXO  characterization  and  classification  problems.  Even  with  optimal  data  quality,  the  estimated 
fit  parameters  cannot  always  be  separated  into  distinct,  non-overlapping  classes  of  UXO  and  non 
UXO-like  objects.  In  fact,  none  of  the  fit  parameters  are  actually  unique  to  UXO  items.  Because 
of  this,  clutter  items  that  physically  resemble  UXO  will  probably  be  misclassified.  Additionally, 
if  the  data  quality  is  not  optimal  the  fitted  parameters  cannot  be  trusted. 
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3.  Demonstration  Design 


3.1  Performance  Objectives 

_ _ Table  1.  Performance  Objectives 


Type  of 
Performance 
Objective 

Primary  Performance 
Criteria 

Expected  Performance  (metric) 

Qualitative 

Ease  of  use 

Minimal  training  required  for  data  processor 
experienced  in  Oasis  montaj™ 

Robustness 

Analysis  flow  not  seriously  interrupted  by  bugs 

Quantitative 

Analysis  time 

<5  min/anomaly  characterization;  0.5  min/anomaly 
document 

False  Positive  Rejection 
Ratio 

0.8 

Location  Accuracy 

0.3m 

Depth  Accuracy 

0.3m 

3.2  Selecting  Test  Sites 

We  selected  the  Former  Badlands  Bombing  Range  because  (i)  we  wanted  to  leverage  the 
ongoing  removal  action  funded  by  the  Corps  of  Engineers,  (ii)  the  contractors,  Corps,  and 
regulatory  personnel  conducting  the  remedial  action  were  cooperative  and  supportive,  and  (iii) 
the  site  presents  a  geophysically  hospitable  environment  in  terms  spatial  registration  issues 
because  of  its  wide-open  nature  and  rolling  terrain.  All  in  all,  the  site  provided  a  realistic 
challenge,  has  an  existing  infrastructure,  and  has  willing  collaborators  conducting  the  OE 
removal  action. 


3.3  Test  Site  History/Characteristics 

The  Former  Badlands  Bombing  Range  is  located  within  the  Oglala  Sioux  Pine  Ridge 
Reservation,  South  Dakota  and  occupies  approximately  341,725  acres  (Figure  8).  Portions  of  the 
Badlands  National  Park  lie  within  the  former  Badlands  Bombing  Range. 

Ownership  and  control  of  land  within  the  boundary  of  the  Former  Badlands  Bombing  Range  is 
highly  complex.  To  simplify,  the  land  is  essentially  owned  by  the  Oglala  Sioux  Tribe  or 
individual  Native  Americans.  Some  of  these  lands  are  leased  to  local  ranchers  or  commercial 
farmers.  Portions  of  the  Badlands  National  Park  that  fall  within  the  boundary  of  the  former 
Range  are  owned  by  the  Oglala  Sioux  Tribe  but  managed  by  the  National  Park  Service.  Right- 
of-entry  access  to  land  within  the  boundary  of  the  range  is  typically  through  the  Oglala  Sioux 
Tribe  and/or  Bureau  of  Indian  Affairs  (BIA).  In  some  instances,  access  can  be  granted  by 
individual  landowners. 
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The  following  site  history  information  was  summarized  from  the  Archives  Search  Report  and 
extracted  verbatim  from  Parson’s  Final  Removal  Action  Work  Plan  dated  April  2004  (Parsons 
2004). 

“In  1942,  the  Rapid  City  Army  Air  Field  (currently  known  as  Ellsworth  Air  Base)  established  a 
bombing  and  aerial  gunnery  range  for  the  purpose  of  training  B-17  combat  crews.  The  land 
utilized  for  the  range  was  within  the  boundaries  of  the  Oglala  Sioux  Pine  Ridge  Reservation.  The 
land  was  acquired  by  four  means:  purchase  or  condemnation  of  privately  owned  land, 
condemnation  lease  of  land  owned  by  the  Oglala  Sioux  Tribe,  a  revocable  permit  obtained  from 
the  BIA,  and  special  use  permit  obtained  from  the  National  Park  Service. 


Figure  8.  Site  map  for  the  Former  Badlands  Bombing  Range. 

By  1947,  a  variety  of  different  aircraft,  including  B-29  and  B-36  bombers,  were  using  the  range 
for  bombing  practice.  In  addition,  fighter  aircraft  were  using  the  site  for  air-to-air  gunnery,  air- 
to-ground  gunnery,  and  dive-bombing  practice.  At  this  time,  radar  bombing  targets  were  also 
located  on  the  site.  In  1958,  the  Air  Force  stopped  using  the  site  for  aerial  gunnery  and  bombing, 
but  bombing  practice  continued  using  radar  scoring.  However,  the  western  portion  of  the  site 
was  retained  by  the  regular  Army  and  the  South  Dakota  Army  National  Guard  for  use  as  an 
impact  area  and  maneuver  area  between  1954  and  1963. 

In  1963,  the  Air  Force  declared  most  of  the  site  as  excess,  but  two  parcels  of  land  were  retained: 
one  for  radar  reflector  targets  and  one  for  a  proposed  fighter  weapons  range.  As  a  result  of  being 
declared  excess,  two  range  clearances  were  conducted:  one  in  1963  and  another  in  1964.  The 
first  clearance  involved  the  land  leased  from  the  Oglala  Sioux  Tribe.  The  second  involved  the 
remainder  of  the  land.  These  clearance  actions  are  thought  to  have  been  only  surface  clearances. 
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The  parcel  retained  as  a  fighter  weapons  range  was  used  as  an  artillery  range  by  the  South 
Dakota  National  Guard  from  about  1960  to  1975.  This  range  had  defined  impact  and  buffer 
areas.  The  parcel  retained  for  radar  reflector  targets  (commonly  referred  to  as  Reflector  City) 
was  used  primarily  for  radar  bomb  scoring.  The  National  Guard  also  established  firing  points  for 
artillery  within  this  range,  and  projectiles  were  fired  into  a  defined  impact  area. 

Training  activities  on  the  site  ceased  and  the  artillery  range  was  swept  in  1975.  The  Department 
of  Defense  transferred  all  of  the  property  back  to  the  Oglala  Sioux  Tribe  and  the  National  Park 
Service,  with  the  exception  of  the  impact  area  of  the  artillery  range.  Ellsworth  Air  Force  Base 
currently  controls  the  impact  area,  also  known  as  the  High  Impact  Area. 

Known  ordnance  used  on  the  range  includes  100-  to  500-pound  sand-filled  bombs;  live  100-  to 
500-lb  bombs;  100-lb  magnesium-filled  photo-flash  bombs;  2.25  inch  sub-caliber  aircraft 
rockets;  2.75-in  folding-fin  aircraft  rockets;  20-  and  75-millimeter  ammunition;  .50-caliber 
ammunition;  3.5-inch  anti-tank  rockets;  and  105-mm  and  8-inch  howitzer  projectiles.” 


3.4  Present  Operations 

Parsons  is  conducting  a  multi-year  OE  removal  action  at  the  BBR  under  contract  to  the  USACE- 
Omaha  District.  Their  objectives  are  to  remove  OE  in  several  areas  including:  existing 
homesteads/working  structures  and  access  roads,  400  and  500-ft  buffers  around  four  bombing 
targets  on  Cuny  Table,  and  a  500  acre  area  in  the  northern  part  of  Cuny  Table  known  as  Gallegos 
table  (Figure  9).  During  the  2005  field  season,  they  acquired  and  interpreted  EM61  Mkll  data 
over  approximately  80+  acres,  using  a  3m-wide  vehicle-towed  array,  at  the  CT-1,  CT-3  and  CT- 
3A  bombing  targets. 

This  demonstration  leverages  Parsons’  work  at  the  CT-3A  bombing  target.  Removing  OE  from 
this  target  is  a  multi-year  task.  The  general  distribution  of  targets  at  the  CT-3  A  target  is  evident 
the  EM61  Mkll  data  presented  in  Figure  10.  In  this  figure,  the  data  at  the  center  of  the  site 
(within  the  large  black  square)  were  acquired  in  2004.  This  demonstration  leveraged  two  grids, 
each  61x61m  (200x200  feet),  directly  north  of  the  bulls-eye  approximately  240m.  Target 
selection  criteria  included  availability,  target  density,  and  degree  of  overlapping  signatures. 

The  landscape  in  the  vicinity  of  the  bombing  target  is  rolling  prairie  (Figure  11). 


13 


Homestead  Selected  for 
Remedial  Action 
Homestead  Not  Selected 
for  Investigation 

Bomb  Target 
Redar  Site 
Gunnery  Target 
CarTargel 
Sector  1  Boundary 
Sector  Boundary 
Road 

Access  Road  Recommended  for 
Removal  of  Subsurface  MEC  Items 

Ranching  Road 
Highway 

Sector  1  QD-Arc  =  21 63'' 

Angsts}  Selected  ror 
Remedial  Act-on 

U$GS  PRG 
Publication  Date  1996 
Npli?:  The  MGFD  for  Secttff  1  l&  the  ICO  lb  General 
Purpose  Mtl  Bcmti  with  Fwo  Mi  03  wtlh  a  maximum 
fragment  range  of  1&63  feet.  Exclusion  zones  wil 
move  arou-iKl  tne  sector  as  the  work  areas 
ninvG- 


N 


3000 


3000  Feat 


U  S.  ARMY  CORPS 

PARSONS  of  engineers 

HUMTSX/I LLE  CENTER 


Gnpg  Matthew 


Grog  MetJhews 


Scuta  Lowry 


Figure  9.  Removal  action  clearance  areas  for  Sector  1;  Former  Badlands  Bombing  Range,  South  Dakota  (Parsons  2004). 
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Figure  10.  Electromagnetic  data  acquired  at  the  CT-3A  area  by  Parsons. 


3.5  Pre-Demonstration  Testing  and  Analysis 

Prior  to  mobilization  of  the  AETC  hardware,  Parsons  acquired  EM61  Mkll  array  data  over  the 
study  area.  Parsons  processed  the  data  and  selected  targets. 
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3.6  Testing  and  Evaluation  Plan 


3.6.1  Demonstration  Set-up  and  Start-up 

The  equipment  were  packed  and  shipped  to  the  site  via  commercial  freight  during  June  2004. 
Leveraged  onsite  infrastructure  included  secure  storage  sheds,  electrical  power,  base  station 
broadcasts  for  Differential  GPS  corrections,  and  an  existing  calibration  grid.  A  variety  of  UXO, 
large  fragments,  and  construction  debris  were  buried  (Figure  12). 
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Figure  12.  Layout  of  the  BBR  geophysical  prove  out.  Burial  depth,  in  meters,  is  shown  above 
each  target  symbol. 

3.6.2  Period  of  Operation 

Parsons  acquired  EM  data  in  July  2005.  Their  data  was  processed  and  delivered  to  AETC  late 
July  2005.  The  NRL-based  magnetometer  and  EMI  systems  underwent  a  shakedown  during 
August  2005.  The  equipment  was  packed  and  shipped  via  commercial  shippers  early  September. 
Tom  Furuya,  AETC  Incorporated,  and  Mark  Howard,  NAEVA  Geophysics  mobilized  to  the  site. 
Magnetic  and  EMI  data  were  acquired  over  the  calibration  area  and  survey  grids  during 
September  12-  16,  2005.  The  equipment  and  field  personnel  demobilized  and  left  the  site 
September  17th.  Preliminary  results  were  presented  at  the  November  2005  and  April  2006  In¬ 
progress  Reviews.  A  detailed  review  was  presented  the  ESTCP  Office  in  June  2006. 


16 


3.6.3  Area  characterized  or  Remediated 


The  survey  area  and  prove  out  grid  covered  1.11  hectares  (2.75  acres).  False-color  data  images  for  the  geophysical  prove  out 
and  survey  grids  are  shown  in  Figure  13  through  Figure  15  respectively. 


Figure  13.  Geophysical  data  acquired  at  the  BBR  prove  out  grid.  The  left  map  shows  electromagnetic  data  and  the  right 
displays  magnetic  signatures. 
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Figure  14.  Electromagnetic  (left)  and  magnetic  (right)  data  acquired  over  two  of  Parsons  61mx61m  (200ft  x  200ft)  grids. 
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Figure  15.  Comparison  of  EM61  Mkll  data  acquired  by  the  man-portable  litter  configuration  (left)  and  Parsons’  three-coil 
array  (right). 
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4.  Performance  Assessment 


4.1  Performance  Criteria 


Table  2.  Criteria  for  this  Demonstration 


Performance 

Criterion 

Description 

Primary  or 
Secondary 

Location 

Accuracy 

The  nominal  radial  distance  error 

Primary 

Depth  Accuracy 

Nominal  absolute  depth  error 

Primary 

False  Positive 
Rejection  Ratio 

Fraction  of  false  alarms  rejected  without  loosing  a  TOI 

Primary 

Ease  of  Use 

Data  analysts’  experienced  with  Oasis  montaj  should  be  able 
to  utilize  the  software  without  manuals  and  without  hands-on 
training. 

Secondary 

Robustness 

Analysis  not  artificially  limited  by  bugs  or  system  design 
flaws 

Secondary 

Analysis  Time 

Times  required  making  a  decision  for  each  anomaly  and 
prepare  required  documentation  supporting  the  decision.  . 

Secondary 

4.2  Performance  Confirmation  Methods 


The  500-plus  anomalies  identified  by  Parsons  were  excavated  as  part  of  their  removal  action 
(Table  3).  This  ground  truth  information,  in  addition  to  digital  photographs,  served  as  the  basis 
for  evaluating  the  discrimination  and  classification  criterions. 


Table  3.  Spreadsheet  used  by  Parsons  to  report  Excavation  Information 


Original  Geophysical 

Survey 

Dig  Results  I 

Grid 

|  AnomalylD 

Northing 

Easting 

1  Ch  3 

Total  Ch 

units 

Response  |  Offset 

Dig  Results 

|  Nomenclature 

Quantity  |  Completeness 

Composition 

Depth  Weight  1  Length 

CT3  4E  ION 

CT3  4E  ION  001 

449536.00 

1320080.00 

i  34 

195 

Mv 

38  :  6 

Munitions  Debris 

RKT,  2.25",  Prac,  SCAR 

1  Warhead 

Aluminum 

5 

21  3 

CT3  4E  ION 

CT3  4E  10N  002 

449537.50 

1320083.00 

1  14 

127 

Mv 

19  i  12 

Munitions  Debris 

RKT,  2.25"  Component 

1  Scrap  Pieces 

Ferrous  Metal 

2 

1  5 

CT3  4E  ION 

CT3  4E  10N  003 

449540.75 

1  320085  00 

!  8 

60 

Mv 

10  j  6 

Munitions  Debris 

RKT,  2.75"  Component 

1 1  Scrap  Pieces 

Mixed  Metals 

3 

1!  5 

CT3  4E  ION 

CT3  4E  10N  004 

449537.00 

1 320091  63 

1  6 

22 

Mv 

11  I  0 

Munitions  Debris 

RKT,  2.75"  Component 

1  Scrap  Pieces 

Aluminum 

2 

1  i  4 

CT3  4E  ION 

CT3_4E_1 0N005 

449522.75 

1320096.75 

:  18 

143 

:m7  "" 

21  6 

Munitions  Debris 

RKT,  2.75"  Component 

1  Scrap  Pieces 

Mixed  Metals 

3 

0.3  5 

CT3  4E  ION 

CT3  4E  10N  005 

449522.75 

1320096.75 

!  18 

143 

Mv 

21  6 

Munitions  Debris 

.50  cal  Debris 

1  Warhead 

Mixed  Metals 

3 

0.3  5 

CT3_4E_10N 

CT3_4E_1 0N006 

449517.75 

1320093.50 

i  6 

31 

Mv 

12  12 

Munitions  Debris 

.50  cal  Debris 

1 1  Warhead 

Ferrous  Metal . 

4 

0.7  3 

CT3_4E_1  ON 

CT3  4E  10N  006 

449517.75 

1320093.50 

i  6 

31 

Mv 

12  12 

Munitions  Debris 

RKT,  2.75"  Component 

1  Scrap  Pieces 

Ferrous  Metal 

4 

0.7:  3 

4.3  Data  Analysis,  Interpretation,  and  Evaluation 

4.3.1  Geophysical  Prove  Out  (GPO) 

The  performance  of  the  litter-carried  and  towed-array  EMI  surveys  of  the  GPO  is  summarized  in 
Figure  16.  The  results  are  presented  as  plots  of  the  dipole  fit  error  as  a  function  of  signal  to  noise 

ratio.  Here,  dipole  fit  error  is  defined  as  e  -  a/  1  -  r 2  x  100,  where  r2  is  the  squared  correlation 


20 


coefficient  between  the  best  model  fit  and  measured  anomaly  data,  and  SNR  is  defined  as 
1 00 a:  peak  anomaly  signal /  var  iance  (noise) .  The  mean  dipole  fit  error  for  the  litter-carried  fit 
results  is  22.7  (standard  deviation  of  20.2)  and  for  the  towed  array  fit  results  is  34.9  (standard 
deviation  of  23.6). 
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Figure  16.  Scatter  plot  of  the  SNR  versus  dipole  fit  error  for  Litter-mode  EMI  data  acquired  over 
the  geophysical  prove  out. 

Size  estimates  from  the  dipole  fits  are  compared  with  measured  UXO  diameters  for  each  of  the 
survey  modes  in  Figure  17.  The  Non-UXO  items  emplaced  within  the  GPO  included  rebar, 
angle  iron,  cylinders,  and  pipes.  For  non-UXO  items,  we  plot  estimated  size  versus  width.  The 
statistics  are  summarized  in  Table  4.  Although  there  is  a  bit  of  scatter  for  the  both  dynamic 
survey  modes,  the  litter  mode  produced  more  accurate  estimates  than  the  towed  array.  The  cued 
interrogation  produced  the  most  accurate  estimates.  As  expected,  we  see  the  accuracy  of  the  size 
estimates  improve  as  the  fit  error  decreases.  Example  data  for  the  litter-carried  survey  data  are 
shown  in  Figure  18.  See  Appendix  A  for  additional  plots  showing  fit  results  for  a  variety  of  fit 
errors. 

Depth  estimates  from  the  dipole  fits  area  compared  with  actual  target  depths  for  each  of  the 
survey  modes  in  Figure  19.  At  this  threshold  of  fit  error  (-32%),  we  observe  a  fair  amount 
scatter  for  all  survey  modes.  Because  the  GPO  was  put  in  a  number  of  years  ago  as  part  of  the 
ongoing  Corps  of  Engineers  cleanup  activities,  we  do  not  know  the  accuracy  of  the  ground  truth 
information.  The  statistics  are  summarized  in  Table  4. 

Target  shape  classification  based  on  dipole  fits  relies  on  the  relationship  of  eigenvalues  of  the 
magnetic  polarizabity  tensor  (here,  referred  to  as  betas).  We  normally  order  the  betas  from 
largest  to  smallest.  Only  fits  with  all  three  (3s  >  0  are  physically  realistic.  Spherical  targets 
should  have  all  three  betas  equal,  cylinders  Pi  >  P2  =  P3,  discs  |3|  =  |3  o  >  P3  and  irregularly  shaped 
objects  |3i  >  P2  >  P3.  Figure  20  shows  scatter  plots  of  the  beta  ratios  (3 7(3 2  and  P2/P3  for  each  of 
the  survey  modes.  For  symmetric  objects,  we  expect  that  P2  ~  P3.  This  expected  pattern  is 
observed  for  the  cued  data  but  not  for  the  two  dynamic  surveys. 
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Table  4.  Error  Statistics  for  Size  and  Depth  Estimates;  GPO;  Fit  Errors  <32% 


Survey  Mode 

%  Mean  Error 
(UXO) 

%  Mean  Error 
(Non-UXO) 

Size  Estimate 

Array 

73 

50 

Litter 

40 

45 

Cued 

10 

13 

Depth  Estimate 

Array 

113 

42 

Litter 

28 

95 

Cued 

39 

32 

EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.31 6228  EM  ARRAY-Actual  vs  fitted  size  -  Fit  Error  It  0.31 6228 


EM  CUED  -  Actual  vs  fitted  size  -  Fit  Error  It  0.31 6228  MAG  -  Actual  vs  fitted  size  -  Fit  Error  It  0.31 6228 


Figure  17.  Scatter  plot  of  the  diameter  versus  estimated  size  for  targets  in  the  GPO  for  each  of 
the  survey  modes.  Targets  with  fit  errors  less  than  -32%  are  included  in  these  plots. 


22 


Size  from  dipole  fit  (mm)  Size  from  dipole  fit  (mm) 


EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.0999999  EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.141421 


EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.223607 


EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.316228 


Figure  18.  Scatter  plot  of  the  diameter  versus  estimated  size  for  different  dipole  fit  error 
thresholds. 
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EM  LITTER  -  Actual  vs  fitted  Depth  -  Fit  Error  It  0.316228  EM  ARRAY  -  Actual  vs  fitted  Depth  -  Fit  Error  It  0.316228 


Figure  19.  Depth  estimates  from  the  dipole  fits  compared  with  actual  target  depth  for  each  of  the 
the  survey  modes.  See  Appendix  A  for  additional  plots  for  different  fit  errors. 
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EM  LITTER  -  Polarizabilities  -  Fit  Error  It  0.316228 


EM  ARRAY  -  Polarizabilities  -  Fit  Error  It  0.316228 


EM  CUED  -  Polarizabilities  -  Fit  Error  It  0.316228 
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Figure  20.  Polarizability  plots  for  GPO  targets  that  have  with  fit  errors  less  than  -32%  for 
different  survey  modes.  See  Appendix  A  for  additional  plots  for  different  fit  errors. 
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4.3.2  CT-3A  Survey  Area 


Excavation  Results  from  CT-3 

473  excavated  anomalies  were  analyzed  as  part  of  this  demonstration.  Eleven  TOI  items  were 
unearthed;  including,  four  20mm  projectiles,  six  2.25”  aluminum  warheads,  and  one  2.75inch 
MK1  ferrous  rocket  warhead  (Figure  21).  The  vast  majority  of  the  remaining  objects  were 
2.75inch-related  munitions  debris  and  50-cal  munitions  debris  (Table  5).  Photographs  of 
commonly-recovered  munitions  debris  are  shown  in  Figure  22. 


Table  5.  Dig  Results  from  CT-3A  Grids 


TOI 

20mm 

4 

2.25inch  aluminum  warheads 

6 

2.75inch  ferrous  rocket  warhead 

1 

Debris 

2.75inch  munitions  debris 

334 

50-cal  munitions  debris 

80 

2.25inch  munitions  debris 

16 

M3  8  bomb  components 

6 

Non-descript  trash,  other,  or  no-contact 

26 

TOTAL 

473 

Figure  21.  Three  types  of  MEC  were  recovered  within  the  CT-3  A  survey  area.  The  include  four 
20mm  projectiles,  six  2.25inch  warheads,  and  one  ferrous  2.75in  Mkl  rocket  warhead. 
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Figure  22.  Photographs  of  non-UXO  objects  recovered  from  the  CT-3A  bombing  site. 
Comparative  Performance  of  Survey  Modes 

The  performance  of  the  two  dynamic  EMI  surveys  is  graphically  presented  in  Figure  23.  As 
before,  the  results  are  presented  as  plots  of  the  dipole  fit  error  as  a  function  of  the  SNR.  The  mean 
fit  error  is  33.9  (standard  deviation  of  27.3)  for  the  litter-carried  data  and  30.0  (standard  deviation 
of  17.5)  for  the  towed  array  data. 
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Signal  to  Noise  Ratio  (dB) 

Figure  23.  Dipole  fit  error  versus  target  SNR  for  Litter  and  Array  survey  modes;  CT-3A  survey 
area. 
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For  baseline  purposes,  cued  data  were  acquired  at  the  marked  locations  in  Figure  24.  When  these 
locations  were  selected  using  Parsons’  towed-array  EMI  data,  we  did  not  know  a  prior  the  nature 
of  the  source.  Efforts  were  made  to  select  isolated  targets  that  possess  a  range  of  amplitudes  and 
spatial  footprints.  EMI  litter-carried  and  towed-array  data  collected  over  a  number  of  the  cued 
targets  is  presented  in  Appendices  C  and  D. 

Performance  of  the  cued  interrogations,  as  measured  by  the  dipole  fit  error  versus  SNR,  is 
considerably  better  than  either  of  the  two  dynamic  surveys.  Figure  25  shows  results  for  each  of 
the  survey  modes.  Including  the  outliers,  the  average  dipole  fit  error  for  cued  data  is  5.1%. 

Figure  26  directly  compares  fit  errors  for  the  cued  versus  dynamic  datasets.  With  one  exception 
(anomaly  #10066),  the  cued  fit  errors  are  substantially  smaller  than  the  corresponding  dynamic 
surveys.  Anomaly  #10066  is  characterized  by  a  2m  by  lm  irregular,  low  amplitude,  non- 
symmetrical  EMI  signature.  When  this  anomaly  was  excavated,  it  was  discovered  that  the 
source  was  20  to  30  inch-size  munitions  debris  fragments. 

Fit  statistics  for  the  cued  targets  within  the  CT-3A  survey  area  are  shown  for  all  survey  modes  in 
Table  6. 


Table  6.  Comparison  of  select  fit  parameters  for  cued  targets. 


ID 

Recovered 

Fit  Error 

Fit  Depth  (m) 

X(Beta's) 

Fit  Error 

Fit  Depth  (m) 

^(Beta's) 

Fit  Error 

Fit  Depth  (m) 

^(Beta's) 

Depth  (m) 

CUED 

CUED 

CUED 

ARRAY 

ARRAY 

ARRAY 

LITTER 

LITTER 

LITTER 

10007 

0.13 

2% 

0.09 

1.04 

4% 

0.10 

1.65 

22% 

0.39 

12.613 

10031 

0.46 

3% 

0.29 

0.72 

19% 

0.31 

0.71 

19% 

0.25 

0.863 

10066 

0.51 

19% 

0.71 

-3.58 

12% 

0.02 

0.47 

35% 

0.8 

10.698 

10069 

0.05 

1% 

0.08 

0.27 

23% 

0.38 

1.71 

11% 

0.34 

1.633 

10078 

0.08 

1% 

0.05 

0.54 

49% 

0.10 

0.88 

43% 

0.08 

1.199 

10079 

0.05 

7% 

0.09 

0.07 

12% 

0.13 

0.15 

25% 

0.43 

0.673 

10080 

0.08 

2% 

0.03 

0.06 

15% 

0.13 

0.27 

32% 

0.35 

0.74 

10108 

0.05 

4% 

0.04 

0.05 

11% 

0.08 

0.07 

32% 

0.36 

0.209 

10113 

0.05 

3% 

0.00 

0.11 

14% 

0.00 

0.14 

19% 

0.03 

0.498 

10120 

0.05 

3% 

0.02 

0.25 

21% 

-0.02 

0.40 

13% 

0.28 

2.174 

10122 

0.05 

2% 

0.06 

0.17 

68% 

0.17 

0.43 

7% 

0.22 

0.623 

10124 

0.05 

1% 

0.04 

0.17 

27% 

0.37 

2.10 

29% 

0.3 

1.495 

10134 

0.05 

2% 

0.04 

0.20 

27% 

0.35 

1.09 

36% 

0.03 

0.355 

10157 

0.05 

5% 

0.07 

0.05 

23% 

0.39 

0.47 

33% 

0.51 

0.866 

10184 

0.05 

3% 

0.11 

0.09 

13% 

0.13 

0.26 

26% 

0.44 

1.334 

10229 

0.05 

11% 

0.05 

0.27 

43% 

0.72 

1.30 

41% 

0.56 

1.504 

10235 

0.05 

3% 

0.08 

0.05 

35% 

0.40 

0.72 

27% 

0.34 

0.461 

10238 

0.08 

2% 

0.18 

0.43 

12% 

0.12 

0.23 

12% 

-0.02 

0.25 

10290 

0.08 

16% 

0.12 

0.25 

31% 

-0.03 

0.08 

30% 

0.33 

3.668 

10293 

0.05 

3% 

0.03 

0.38 

9% 

0.22 

2.52 

6% 

0.2 

1.847 

10294 

0.05 

11% 

0.00 

0.04 

23% 

0.36 

0.30 

14% 

0.51 

0.97 

AVERAGE 

0.10 

5% 

0.10 

0.08 

23% 

0.21 

0.76 

24% 

0.32 

2.13 

Std  Deviation 

0.13 

5% 

0.15 

0.88 

15% 

0.19 

0.71 

11% 

0.20 

3.28 
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Figure  24.  False  color  map  of  EM61  towed-array  data  showing  the  locations  of  targets  selected 
for  cued  interrogations. 
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Figure  25.  Scatter  plot  of  dipole  fit  error  versus  SNR  for  Litter,  towed-Array,  and  Cued 
interrogations  for  common  targets  in  the  CT-3A  survey  area. 
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Figure  26.  Scatter  plots  of  inverted  fit  results  for  cued  versus  dynamic  deployments  -  common, 
native  targets  within  the  CT-3A  survey  area.  Left  -  cued  versus  litter  mode.  Right  -  cued  versus 
towed  array. 
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Parameter  Estimates  from  the  CT-3A  Grids 

Figure  27  presents  scatter  plots  of  the  inverted  model  parameters  for  the  litter-carried  EMI  data. 
Black  dots  identify  parameter  values  for  individual  anomalies.  Recovered  TOI  objects  are 
identified  using  colors  and  symbols. 

The  estimated  burial  depths  are  not  accurate  for  a  majority  of  the  small  shallow  objects.  Figure  28 
presents  scatter  plots  of  the  burial  depth  as  recorded  during  excavation  versus  the  estimated  depth 
for  a  variety  of  fit  errors  and  all  EMI  survey  modes.  We  note  that  (i)  very  few  targets  possess  fit 
errors  less  than  roughly  10%  for  the  dynamic  surveys  (shown  in  red  and  blue  symbols),  and  (ii)  the 
results  are  very  poor  for  targets  buried  less  than  20cm  below  ground  level.  The  litter-carried  and 
towed-array  systems  performed  similarly.  On  average,  estimated  depths  are  within  a  5- 10cm  for 
fit  errors  less  than  10%,  but  the  RMS  error  is  ~20cm  for  both  survey  modes,  which  amounts  to 
approximately  1/3  of  the  target/coil  separation  (40cm  standoff  +  10cm  burial).  For  these  data,  the 
fit  depth  error  is  not  a  strong  function  of  fit  quality.  The  mean  and  RMS  depth  error  for  all  of  the 
survey  data  are  plotted  as  functions  of  dipole  fit  error  in  Figure  29  and  Figure  30.  For  each  plotted 
fit  error  value,  the  mean  and  RMS  error  are  calculated  over  the  band  ±0.003,  which  typically 
includes  10-15  samples. 

There  is  evidence  that  the  data  density  and  observed  signal  strength  may  not  be  adequate  to 
support  reliable  inversion  for  many  of  the  targets,  even  if  the  fit  quality  is  high.  In  such  case  the 
dipole  model  can  reproduce  the  data,  but  there  is  not  enough  data  to  constrain  the  fit  to  the  correct 
target  parameters.  Figure  31  presents  a  scatter  plot  showing  the  number  of  points  within  individual 
anomaly  footprints  that  are  greater  than  20mV  versus  dipole  fit  error.  As  shown  in  this  figure,  the 
vast  majority  of  anomalies  possess  fewer  than  20  measurements  with  values  greater  than  20mV. 
Figure  32  presents  a  similar  plot  comparing  the  number  of  points  within  the  anomaly  with  values 
greater  than  20mV  with  the  depth  error.  As  before,  depth  errors  decrease  as  the  number  of  high 
amplitude  measurements  increase. 

Estimated  size  is  plotted  against  peak  signal  and  fit  depth  in  Figure  33.  Results  for  both  EMI 
surveys  and  the  magnetic  survey  are  shown.  Black  symbols  are  used  to  show  the  fitted  features  for 
all  anomalies  found  in  the  CT-3A  survey  area.  TOI  items  are  superimposed  using  color  symbols. 
Red  identifies  the  recovered  2.25  inch  aluminum  warheads  and  20mm  projectiles,  and  blue  marks 
the  ferrous  2.75inch  MK1  rocket  warhead. 

With  regards  to  location  accuracy,  the  mean  location  error  for  recovered  20mm  projectiles, 
2.25inch  warheads,  and  2.75inch  warheads  was  20.7cm  with  a  standard  deviation  of  7.6cm. 
Including  the  clutter,  the  average  location  error  was  26.3cm  with  a  standard  deviation  of  16.8cm. 
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Figure  27.  Feature  space  plots  for  data  acquired  in  the  CT-3A  survey  area  (black  dots).  The 
recovered  MEC  items  are  overlain  and  color  coded  for  reference.  Appendix  B  presents  results 
for  a  variety  of  fit  errors. 
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BBR  CT3  EM  -  Actual  vs  fitted  depth  -  Fit  Error  LE  0.031 6226  BBR  CT3  EM  -  Actual  vs  fitted  depth  -  Fit  Error  LE  0.07071 06 


Actual  depth  to  object  (m) 

BBR  CT3  EM  -  Actual  vs  fitted  depth  -  Fit  Error  LE  0.0999999 


Actual  depth  to  object  (m) 

BBR  CT3  EM  -  Actual  vs  fitted  depth  -  Fit  Error  LE  0.223607 


BBR  CT3  EM  -  Actual  vs  fitted  depth  -  Fit  Error  LE  0.316228 
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Figure  28.  Reported  versus  estimated  depth  for  CT-3A  targets  as  a  function  of  survey  modes  and 
fit  errors. 
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Figure  29.  Left  -  scatter  plot  of  dipole  fit  error  versus  depth  error  for  the  litter-carried  EMI 
system.  Right  -  RMS  and  mean  depth  errors  binned  over  discrete  fit  errors. 


Figure  30.  Left  -  scatter  plot  of  dipole  fit  error  versus  depth  error  for  the  towed  Array  EMI 
system.  Right  -  RMS  and  mean  depth  errors  binned  over  discrete  fit  errors. 
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Number  Points  Gt  20mV  Number  Points  Gt  20mV 


Figure  31.  Left  -  scatter  plot  showing  the  number  of  points  within  individual  anomaly  footprints 
that  are  greater  than  20mV  versus  dipole  error  for  the  litter-carried  EMI  system.  Right  -  RMS 
and  mean  depth  errors  binned  over  discrete  fit  errors. 
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Figure  32.  Left  -  scatter  plot  showing  the  number  of  points  within  individual  anomaly  footprints 
that  are  greater  than  20mV  versus  depth  error  for  the  litter-carried  EMI  system.  Right  -  RMS 
and  mean  depth  errors  binned  over  discrete  fit  errors. 
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Figure  33.  Plots  showing  the  peak  signal  and  fitted  depth  versus  fitted  size  for  the  towed  Array 
EMI  system  (top  left),  litter-carried  EMI  system  (top  right),  and  magnetic  sensors  (bottom).  TOI 
items  are  superimposed  using  color  symbols.  Red  identifies  all  the  recovered  TOIs  (2.25  inch 
aluminum  warheads  and  20mm  projectiles),  and  blue  marks  the  ferrous  2.75inch  MK1  rocket 
warhead. 

As  a  whole,  data  acquired  in  a  grid  during  the  cued  interrogations  was  better  characterized  by 
dipole  models  than  the  standard  towed-array  or  litter-carried  EMI  surveys  (Figure  34).  As  a  result, 
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the  residual  dipole  fit  errors  are  significantly  smaller.  We  attribute  the  performance  improvements 
to  the  fact  that  (i)  the  coil  is  slightly  closer  to  the  ground  in  the  cued  interrogations,  and  (ii)  the 
relative  spatial  positions  are  precise  (errors  measured  in  millimeters). 

BBR  CT3  EM  -  EM61  Fit  Quality 
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Figure  34.  Scatter  plot  showing  the  maximum  signal  (in  mV)  versus  the  dipole  fit  error  for  all 
target  analyzed  within  the  CT-3A  survey  area. 

Receiver  Operating  Characteristic  (ROC)  Curves 

For  these  data,  target  classification  is  problematical  for  three  reasons.  First,  the  GPO  data 
available  for  training  is  severely  limited.  The  objects  buried  in  the  geophysical  prove  out  do  not 
adequately  represent  objects  that  were  recovered  within  the  CT-3A  survey  area.  There  was,  for 
example,  no  site-specific  clutter  or  2.25inch  aluminum  warheads  emplaced  in  the  GPO.  The  GPO 
was  designed  and  constructed  by  the  Corps  of  Engineers  to  aid  in  detection  surveys,  not 
specifically  to  aid  discrimination  investigations.  Second,  the  fitted  features  of  the  buried  UXO  and 
simulants  within  the  GPO  do  not  cluster  or  indicate  axial  symmetry  (Figure  35).  This  is  not 
surprising,  however,  given  the  dipole  fit  errors  of  over  20%  (Figure  16).  Third,  the  number  of 
recovered  TOIs  within  the  CT-3A  survey  area  is  severely  limited.  The  TOIs,  defined  in  Parsons’ 
Demonstration  Plan  as  items  larger  than  40mm  projectiles,  include  six  2.25inch  aluminum 
warheads,  and  one  2.75inch  ferrous  rocket  warhead. 

Performance  results  for  these  TOIs  are  presented  using  receiver  operating  characteristic  (ROC) 
curves  in  Figure  36  through  Figure  38.  We  selected  features  that  are  intrinsic  to  the  target;  namely, 
the  magnetic  polarizabilities  and  their  ratios  (Table  7). 
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Figure  35.  Scatter  plot  showing  polarizability  ratios  for  objects  buried  in  the  GPO.  There  were 
no  examples  of  2.25inch  aluminum  warheads.  Fit  results  for  the  2.75inch  nose  simulants, 
rockets,  and  rocket  bodies  are  plotted  in  red. 
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Table  7.  Selected  Model  Parameters,  or  features,  used  as  input  to  the  GLRT  classifier 


Data  Type 

Features  selected  for  use  by  the  GLRT 
classifier 

Magnetic  data 

Moment,  Solid  Angle,  Depth 
or 

Solid  Angle 

EMI  data 

PLP2,  P3 
or 

PI,  P2,  P3,  P1/P2,  P2/P3 

Figure  36  shows  results  for  the  case  in  which  training  and  testing  was  restricted  to  CT-3- 
recovered  UXO  larger  than  40mm  (as  per  the  site  Demonstration  Plan).  Here,  this  includes  the 
2.25inch  aluminum  warheads,  and  the  2.75inch  rocket.  Marginal  to  poor  discrimination 
capabilities  is  observed  for  both  survey  modalities. 

Figure  37  shows  results  for  the  case  in  which  the  classifier  was  trained  using  UXO  items  from 
the  GPO  (green)  and  the  CT-3A  area  (black  and  blue),  and  tested  on  all  CT-3-recovered  UXO  - 
including  the  20mm  rounds.  No  discrimination  capability  is  observed  for  both  survey  modes 
regardless  of  training. 
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LITTER  Train  On:  2.25in  &  2.75in  Rkt 

Features:  Polarizabilities+Ratios 
Test  For:  2.25in  &  2.75in 


ARRAY  Train  On:  2.25in  &  2.75in  Rkt 

Features:  Polarizabilities  +  Ratios 
Test  For:  2.25in,  &  2.75in 


Figure  36  .  Receiver  Operating  Characteristic  (ROC)  curve  for  EMI  data  acquired  in  the  CT-3A 
area  and  classified  according  to  the  color-coded  legend. 


LITTER  Train  On:  Sampling  20mm,  2.25in,  &  2.75in 
Features:  Polarizabilities  from  CT3  Area 
Test  For:  20mm,  2.25in,  &  2.75in 

LITTER  Train  On:  UXO  from  Prove  Out 

Features:  Polarizabilties  from  Prove  Out 
Test  For:  All  20mm,  2.25in,  &  2.75in 

ARRAY  Train  On:  Sampling  20mm,  2.25in,  &  2.75in 
Features:  Polarizabilities  from  CT3  Area 
Test  For:  All  20mm,  2.25in,  &  2.75in 


0.0  0.2  0.4  0.6  0.8  1.0 

Pfa 

Figure  37.  Receiver  Operating  Characteristic  (ROC)  curve  for  EMI  data  acquired  in  the  CT-3A 
area  and  classified  according  to  the  color-coded  legend.  As  shown  here,  no  discrimination 
capability  is  realized  if  the  objective  is  to  discriminate  all  20mm,  2.25inch  warheads,  and 
2.75inch  rockets  and  warheads  from  the  munitions  debris  that  is  present. 

Figure  38  shows  results  for  three  different  discrimination  objectives.  The  black  curve  was 
generated  by  training  and  testing  for  20mm,  2.25inch,  and  2.75inch  TOI.  The  red  curve  was 
generated  training  and  testing  for  2.25  and  2.75inch  TOI  only.  The  green  curve  shows 
performance  results  if  the  objective  is  to  only  discriminate  the  2.75in  rocket  from  the  munitions 
debris.  As  seen  in  this  figure,  it  is  possible  to  differentiate  the  ferrous  2.75inch  rocket  warhead 
from  the  multitude  of  munitions  debris,  but  not  the  smaller  2.25inch  aluminum  warhead.  This 
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result  is  not  unexpected  given  the  large  measured  response  of  the  2.75in  rocket  relative  to  that 
generated  by  the  munitions  debris  (Figure  33). 


LITTER  Train  On:  2.25in  &  2.75in 

Features:  Polarizabilities+Ratios 
Test  For:  2.25in  &  2.75in 


LITTER  Train  On:  20mm,  2.25in,  &  2.75in 
Features:  Polarizabilities 
Test  For:  20mm,  2.25in,  &  2.75in 

LITTER  Train  On:  2.75in  Rkt 

Features:  Polarizabilities 
Test  For:  2.75in  Rkt 


Figure  38.  Receiver  Operating  Characteristic  (ROC)  curve  for  EMI  data  acquired  in  the  CT-3A 
area  and  classified  according  to  the  color-coded  legend. 


Because  the  number  of  TOIs  is  small,  the  ROC  curves  shown  in  Figure  36  through  Figure  38  allow 
the  GLRT  classifier  to  train  and  test  on  the  entire  population.  Because  of  this,  the  performance 
results  may  not  be  indicative  of  future  performance.  Figure  39  and  Figure  40  present  performance 
results  using  a  simpler,  alternate  metric  based  on  the  fitted  size.  For  EMI,  the  fitted  size  is  defined 
as  the  cubed  root  of  the  sum  of  the  polarizability  response  coefficients.  For  magnetic  data,  it  is 
scaled  magnetic  moment. 


Figure  39  is  for  the  case  where  the  TOIs  only  includes  ferrous  2.75inch  rocket  warheads  and 
prioritized  according  to  the  fitted  size.  In  Figure  40,  the  TOIs  include  the  2.75inch  and  2.25  SCAR 
warheads.  For  comparison  purposes,  results  based  on  the  peak  measured  signal  strength  (instead 
of  the  fitted  size)  is  shown  using  dashed  lines. 
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LITTER 

Sort  by:  Fitted  Size  (solid) 

Sort  by:  Signal  Strength  (dashed) 

ARRAY 

Sort  by:  Fitted  Size  (solid) 

Sort  by:  Signal  Strength  (dashed) 


Test  For:  2.75in  Rocket 
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Figure  39.  Receiver  Operating  Characteristic  (ROC)  curve  for  litter-carried  and  towed-array 
EMI  data  based  on  fitted  size;  TOI:  2.75inch  rocket  only. 


LITTER  Sort  by:  Fitted  Size  (solid) 

Sort  by:  Signal  Strength  (dashed) 


ARRAY  Sort  by:  Fitted  Size  (solid) 

Sort  by:  Signal  Strength  (dashed) 


Test  For:  2.25in  &  2.75in  Rocket 
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Figure  40.  Receiver  Operating  Characteristic  (ROC)  curve  for  litter-carried  and  towed-array 
EMI  data  based  on  fitted  size;  TOI:  2.25inch  &  2.75inch  rocket  warheads. 

Figure  41  shows  discrimination  performance  results  for  the  magnetic  sensor  based  on  fitted  size 
and  observed  magnitude  (analytic  signal).  For  magnetometry  data,  we  do  no  consider  the  2.25inch 
aluminum  warheads  or  20mm  projectiles  to  be  legitimate  discrimination  targets  at  this  site  for  two 
reasons.  First,  because  the  2.25inch  warheads  are  aluminum,  they  should  not  produce  an  invertible 
magnetic  signature.  Second,  because  the  site  is  littered  with  small  munitions  debris,  we  cannot 
confidently  say  that  the  low  amplitude  responses,  ranging  from  4-10nT,  observed  in  the  vicinity  of 
the  20mm  projectiles  are  caused  by  the  subject  item.  By  way  of  example,  there  are  invertible 
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magnetic  signatures  within  0.25m  of  five  of  the  six  2.25inch  aluminum  warheads.  Because  of 
these  factors,  we  are  limited  to  looking  for  a  single  item,  the  ferrous  2.75inch  rocket  warhead, 
within  a  sea  of  small  munitions  debris. 


MAG 

Sort  by:  Peak  Signal  (solid) 

Sort  by:  Signal  Strength  (dashed) 


Test  For:  2.75in  Rocket 


0.0  0.2  0.4  0.6  0.8  1.0 
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Figure  41.  Receiver  Operating  Characteristic  (ROC)  curve  for  Mag  data  acquired  in  the  CT-3A 
area  and  prioritized  according  to  the  fitted  magnetic  moment. 

Table  8  expresses  the  results  in  a  slightly  different  manner.  Here,  we  show  the  percentage  of 
targets  that  possess  fitted  sizes  and  signal  strengths  that  are  50%  or  80%  that  of  the  subject  TOI. 
The  results  shown  below  include  all  fitted  anomalies  -  we  have  not  segmented  based  on  the  quality 
of  the  fit.  As  a  result,  the  fitted  size  performance  results  are  degraded  relative  to  signal  strength  by 
inaccurate  inversion  results  due  to  overlapping  signatures  and  spatial  registration  errors. 

Table  8.  Percent  of  false  positives  that  possess  fitted  sizes  below  specific  thresholds 


TOI  Definition 

Survey 

Mode 

%  Fit  Size 

Vi  Signal 
Strength 

0.8*Fit 

Size 

0.8*Signal 

Strength 

2.75inch  Rocket 
Only 

LITTER 

39% 

97% 

80% 

97% 

ARRAY 

55% 

85% 

84% 

96% 

MAG 

68% 

96% 

79% 

97% 

2.25inch  & 
2.75inch 
warheads 

LITTER 

3% 

39% 

26% 

62% 

ARRAY 

1% 

29% 

7% 

52% 

42 


4.3.3  Qualitative  Metrics 

UX-Analyze  characterizes  and  classifies  individual  anomalies.  Steps  taken  within  the 
characterization  phase  include  identifying  anomalies,  selecting  the  spatial  footprint  of  the 
anomaly,  and  reviewing  the  results.  The  most  labor  intensive  portion  of  the  characterization 
phase  is  identifying  the  spatial  extent  of  each  anomaly  and  editing  individual  samples  that  are 
inconsistent.  UX-Analyze  provides  graphical  presentations  and  appropriate  tools  to  assist  the 
analyst.  Once  completed,  individual  anomalies  are  inverted  manually  or  in  batch  mode.  The 
output  of  the  characterization  phase  is  a  list  of  model  parameters  for  each  anomaly  which  is  then 
submitted  to  the  classifier.  Steps  taken  during  the  classification  phase  include  creating  or 
updating  a  library  for  training  purposes,  selecting  and  calling  the  classifier. 

UX-Analyze  adheres  to  the  look  and  feel  of  Oasis  montaj  and,  as  such,  is  quickly  mastered  by 
experienced  Oasis  montaj  users.  It  is  easy  to  use  and  includes  an  on-line  help  menu. 

Researchers  at  Weston,  Parsons,  and  the  Naval  Research  Laboratory  have  used  UX-Analyze 
during  recent  months  in  their  projects  without  incident.  During  the  course  of  this  demonstration, 
Oasis  montaj  underwent  a  number  of  major  upgrades  that  affected  the  performance  and 
robustness  of  UX-Analyze.  A  large  number  of  technical  stumbling  blocks  were  created  when 
Oasis  montaj  package  migrated  from  version  5  to  version  6.  UX-Analyze  was  integrated  into  the 
official  build  of  Oasis  montaj  in  Version  6.2,  which  allows  it  to  be  licensed,  installed,  and 
managed  using  Geosoft’s  on-line  licensing  system.  Fixes  to  a  normalization  issue  within  the 
GLRT  classification  routines  were  completed  in  December  2006.  Fixes  to  a  figure  of  merit 
calculation  were  completed  in  March  2007.  All  known  coding  inconsistencies  and  bugs  have 
been  corrected  as  of  this  report.  The  current  Oasis  montaj  release  is  6.4,  and  we  do  not 
experience  systematic  difficulties  while  characterizing  and  classifying  these  magnetic  and  EMI 
data. 


4.3.4  Discussion 

The  objective  of  the  Corps  of  Engineers  ordnance  and  explosive  (OE)  removal  action  is  to 
remove  all  items  ranging  between  a  40mm  projectile  and  a  5001b  bomb  at  depths  of  less  than  11- 
diameters  of  the  object,  to  a  clearance  depth  of  four  feet.  We  analyzed  data  from  part  of  the 
bombing  target  with  the  hope  of  identifying  particular  TOIs  that  could  be  discriminated  from  the 
plethora  of  munitions  debris  that  is  present  at  this  locale.  Excavation  results  produced  no  intact 
UXO  within  the  study  area.  Recovered  TOIs  within  the  two,  CT-3-bombing  grids  studied  here 
included  four  20mm  projectiles,  six  2.25inch  aluminum  SCAR  warheads,  and  one  ferrous 
2.75inch  rocket  warhead. 

Discriminating  the  20mm  projectiles  and  2.25inch  aluminum  warheads  from  the  plethora  of 
munitions  debris  responses  based  on  their  inverted  source  parameters  proved  challenging.  Even 
though  the  litter-based  EMI  sensor  was  carried  along  transects  spaced  0.5m  apart,  the  majority  of 
signatures  were  observed  on  only  a  single  profile.  As  a  result  of  the  limited  spatial  views  of  the 
target  (e.g.,  we  did  not  illuminate  the  targets  from  all  angles  due  to  the  anomalies  small  footprint) 
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and  the  low  amplitudes,  the  EMI  inversion  results  do  not,  as  a  rule,  provide  accurate  size,  shape, 
and  depth  estimates  even  when  the  dipole  fit  error  is  small. 

As  measured  by  the  dynamic  survey  platforms,  the  anomalies  were  predominantly  spatially 
limited  and  could  not  be  well  represented  by  a  dipole  model.  The  majority  have  dipole  fit  errors 
much  greater  than  10%  (Figure  34)  and  does  not,  therefore,  support  discrimination  decisions 
based  on  inverted  source  parameters.  Previous  experience  suggests  that  the  major  factors 
limiting  field  performance  are  positioning  errors,  background  noise,  overlapping  signatures,  and 
data  density.  Systematic  or  random  noise  in  the  sensor  readings  caused  by  these  factors  linearly 
affects  the  inverted  magnetic  polarizabilities  and  degrades  discrimination  performance.  Figure 
42  shows  the  relationship  between  errors  in  target  parameter  estimates  (|3s)  and  the  dipole  fit 
error  based  on  Monte-Carlo  simulations  (Bell,  2005).  In  order  to  get  target  parameters  with 
sufficient  accuracy  (<  30-50%)  to  support  target  classification,  we  need  dipole  fit  errors  of  5%  or 
less. 


Figure  42.  Beta  error  versus  dipole  fit  error  from  Monte  Carlo  simulations. 

In  stark  comparison  to  the  litter-carried  and  towed-array  EMI  data,  the  gridded  EMI  data 
acquired  in  a  cued  fashion  was  well  represented  by  a  dipole  model.  For  targets  within  the  CT-3A 
bombing  target,  the  mean  dipole  fit  error  was  5%  compared  to  >23%  for  the  dynamic  surveys. 
Furtheremore,  with  the  exception  of  one  horizontal  37mm  buried  at  30cm,  the  inverted  shape 
parameters  for  symmetric  targets  in  the  GPO  appear  symmetric. 

A  significant  number  of  the  munitions  debris  can  be  screened  by  setting  appropriate  thresholds  if 
the  stakeholders  and  regulators  set  specific  TOIs  for  the  site.  If  the  TOI  is  restricted  to  2.75inch 
ferrous  warheads  and  larger  items,  for  example,  these  data  indicate  that  as  many  as  97%  of  the 
excavate  munitions  debris  could  have  been  safely  left  in  the  ground.  Alternatively,  if  the  TOIs 
include  2.25inch  and  larger  objects,  30%  to  60%  of  the  munitions  debris  can  remain  depending 
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on  whether  the  amplitude  threshold  is  set  to  50%  or  80%  of  that  observed  for  the  subject 
2.25inch  aluminum  warheads. 

In  addition  to  analyzing  individual  anomalies  that  were  selected  by  Parsons  using  their  towed 
array  EMI  data  for  ground  truthing,  we  looked  for  additional,  comparable  targets  in  the  litter- 
carried  data  that  might  be  evident  due  to  the  more  conservative  transect  spacing.  The  towed 
array  data  was  acquired  along  transect  spaced  one  meter  apart,  and  the  litter  carried  data  was 
acquired  along  transect  spacing  that  were  nominally  0.5m  apart.  We  identified  over  340 
additional  targets  in  the  litter-carried  EMI  data  set.  The  extra  anomalies  are  characteristically 
small,  having  peak  responses  of  20mV  or  less,  which  is  similar  to  that  observed  for  the  20mm 
projectiles  and  small  munitions  debris,  and  were  typically  located  between  the  towed-arrays 
transects.  Although  these  additional  targets  were  not  unearthed  as  part  of  the  Corps  of  Engineers 
removal  action,  it  is  clear  by  the  number  of  extra  anomalies  and  by  the  high  number  of  single- 
transect  anomalies  (in  the  towed-array  data)  that  small  items  are  preferentially  detected  along 
survey  lines  compared  to  between  the  transects. 

Although  largely  complimentary,  the  magnetic  survey  detected  a  few,  noteworthy  potential 
source  targets  that  were  not  readily  observed  in  the  EMI  data  sets.  First,  two  2.75inch  rockets  in 
the  GPO  (Figure  13)  possess  easily  identifiable  magnetic  dipoles  but  no  EMI  response.  Second, 
two  large  anomalies  are  observed  within  the  CT-3A  survey  grids  (Figure  43).  The  source  of 
these  later  anomalies  is  not  known,  but  the  coordinates  and  fit  information  has  been  passed  on  to 
the  Omaha  Corps  of  Engineers. 
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Figure  43.  Black  arrows  identify  two  large  magnetic  anomalies  that  do  not  have  corresponding 
EMI  signatures. 
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5.  Cost  Assessment 


5.1  Cost  Reporting 

This  demonstration  focuses  on  characterizing  and  classifying  anomalies  observed  in  magnetic  and 
electromagnetic  data.  It  encompasses,  therefore,  only  a  subset  of  costs  that  are  typically  associated 
with  acquisition  demonstrations.  The  relevant  cost  categories  for  data  analysis  are  shown  in  Table 
9. 


Table  9.  Cost  categories  and  details 


Cost  Category 

Details 

Sub  Category 

Time 

(hours) 

Costs* 

($) 

Re-leveling  &  digital 
filtering  prior  to 
inversion 

EM61  Towed  Array 

8 

800 

Pre-processing 

EM61  Litter 

40 

4000 

EM61  Cued 

4 

400 

Mag 

16 

1600 

Culling  anomaly 
signatures  from  site 
database 

EM61  Towed  Array 

16 

1600 

Anomaly 

EM61  Litter 

24 

2400 

extraction 

EM61  Cued 

0 

0 

Mag 

16 

1600 

Data 

Inverting  anomaly  data 

EM61  Towed  Array 

16 

1600 

characterization 

for  model  parameters  & 

EM61  Litter 

16 

1600 

and 

performing  feature-based 

EM61  Cued 

4 

400 

classification 

classification 

Mag 

16 

1600 

Documentation 

Preparing  Dig  Sheets 
using  UX- Analyze 
functions 

All  data  types 

-100  per 
minute 

— 

Reporting 

Technical  Report 

— 

350 

35,000 

TOTAL 

52,600 

*assumes  an  fully-loaded  labor  rate  of  $100  per  hour 


6.  Implementation  Issues 


6.1  End-Users  Issues 

The  primary  end-users  of  this  data  analysis  technology  will  likely  be  high-end  geophysical  service 
providers,  technically  orientated  DoD  oversight  personnel,  and  government-sponsored  researchers. 
In  order  to  be  successfully  transitioned  to  the  production  community,  this  technology  must  not 
only  be  accepted  by  the  Environmental  Protection  Agency  and  the  Corps  of  Engineers,  it  must 
initiate  a  change  in  the  requested  deliverables  regarding  data  analysis  for  UXO  concerns. 
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Currently,  the  Corps  of  Engineers  Data  Item  Description  (DID)  OE-005-05.01,  which  details  the 
required  information  for  each  anomaly,  does  not  ask  for  estimates  of  the  targets’  features  (viz., 
such  as  target  depth,  apparent  size,  orientation,  etc.)  It  only  requests  the  amplitude  of  the  anomaly 
and  a  prioritized  ranking.  This  program  explored  the  methods  and  logic  involved  with  creating  a 
prioritized  dig  list. 
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8.  Points  of  Contact 


ESTCP 

Anne  Andrews  ESTCP 

901  North  Stuart  Street 
Suite  303 

Arlington,  VA  22203 


Katherine  Kaye  HydroGeologic,  Inc. 

1155  Herndon  Parkway 
Suite  900 

Herndon,  VA  20170 


Tel:  703-696-3826  Program  Manager 
Fax:  703-696-21 14  UXO  Thrust  Area 
Anne.Andrews@osd.mil 


Tel:  410-884-4447  Program  Assistant 
Fax:  703-478-0526  UXO  Thrust  Area 
jef@hgl.com 


SAIC  (formerly  AETC  Incorporated) 

Dean  Keiswetter  SAIC 

120  Quade  Drive 
Cary,  NC  27513 


Tel:  919-653-0215x103  PI 
Fax:  919-653-0219 
keiswetterd  @  saic.com 


Tom  Bell 


SAIC 

1225  Jefferson  Davis  Highway 
Suite  800 

Arlington,  VA  22202 


Tel:  703-413-0500x254  Co-PI 
Fax:  703-413-0512 
bellth@saic.com 


Tom  Furuya 


SAIC 

120  Quade  Drive 
Cary,  NC  27513 


Tel:  919-653-0215x104  Data  Analyst 
Fax:  919-653-0219 
tfuruy  at  @  s  aic .  com 


Parsons 

Greg  Van  Parsons 

1700  Broadway,  Suite  900 
Denver,  CO  80290 


Tel:  303-764-1927 
Fax:  410-278-1589 
Greg.Van@parsons.com 


US  Army  Corps  of  Engineers  Omaha  District 

Richard  Grabowski  US  Army  Corps  of  Engineers  Tel:  402-221-7784 

Attn:  CENWO-ED-GG  (Grabowski)  Fax:  410-278-1589 
106  South  15th  Street 

Omaha,  NE  68102-1618  RichardJ.Grabowski@usace.army.mil 
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Appendix  A  -  Parameter  Estimates  from  the  Prove  Out 


Appendix  A  -  5 1 


Size  from  dipole  fit  (m)  Size  from  dipole  fit  (m) 


Size 


MAG  -  Actual  vs  fitted  size  -  Fit  Error  It  0.0999999  MAG  -  Actual  vs  fitted  size  -  Fit  Error  It  0.141 421 


MAG  -  Actual  vs  fitted  size  -  Fit  Error  It  0.223607 


MAG  -  Actual  vs  fitted  size  -  Fit  Error  It  0.31 6228 


Appendix  A  -  52 


Size  from  dipole  fit  (mm)  Size  from  dipole  fit  (mm) 


EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.0999999  EM  LITTER  -  Actual  vs  fitted  size  -  Fit  Error  It  0.141421 


Appendix  A  -  53 


Size  from  dipole  fit  (mm)  Size  from  dipole  fit  (mm) 


EM  CUED  -  Actual  vs  fitted  size  -  Fit  Error  It  0.0999999 


EM  CUED  -  Actual  vs  fitted  size  -  Fit  Error  It  0.141421 
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Size  from  dipole  fit  (mm)  Size  from  dipole  fit  (mm) 
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Fitted  Depth  (m)  Fitted  Depth  (m) 
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Fitted  Depth  (m)  Fitted  Depth  (m) 
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Fitted  Depth  (m)  Fitted  DePth  (m) 
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Fitted  Depth  (m)  Fitted  Depth  (m) 
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Chi2  Error  Size  (mm) 
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Appendix  B  -  Parameter  Estimates  from  the  CT-3A  Survey  Area 
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Appendix  C  -  Litter  mode  EMI  data  for  Cued  Targets 
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Appendix  D  -  Towed  Array  EMI  data  for  Cued  Targets 
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